Monday, October 20, 2014
Introduction
- Why does causality matter for policy making?
- Why do we like experiments?
- Why do we hate experiments?
- Turning a weakness into a strength
- What does policy analysis look like in the real world?
Causality Matters
- But politics has a way of trying to make it seem so
But it's not everything
- Decisions have to be made with best evidence available
- Ruling out alternative explanations generally comes with increased costs
- Scientific process downplays the subjective judgments of domain experts in favor of more expensive objective measurement
- May
Experiments, Yay!
Experiments Do
- Provide solid internal reliability when done well
- Provide idealized estimates of impact
- Often test interventions in a best-case scenario
- Provide clean data with transparent analyses to draw conclusions
Experiments, Boo!
Experiments Do Not
- Have external reliability. Does the lab resemble the real world?
- Often ignore the importance of precise implementation and often fail to measure the variability in implementation that will occur.
- Do not help understand the reasons program attrition might occur
- Are limited in their ability to test the variability in response to treatment under different conditions – recruiting and retaining subjects is $$
Experiments in Policy
- A randomized controlled trial (RCT) in policy is rarely possible and rarely desirable
- RCTs require a level of control and authority that regulators rarely possess
- RCTs require a level of monitoring and tracking that may not be appropriate for a governmental entity
- Recruiting and retaining subjects is expensive and can substantially add to the bill of a policy
- Lack of external validity makes it hard to say "Program X worked in the trial, let's go to scale."
An Aside on the Policy Process
John Conway's Game of Life
- Simple rules in large systems create emergent properties that are complex and unpredictable
- A simple example is John Conway's Game of Life, played on a two-dimensional grid
- Any live cell with fewer than two neighbors dies
- Any live cell with two or three neighbors lives
- Any live cell with more than three live neighbors dies
- Any dead cell with exactly three live neighbors becomes a live cell
- These simple patterns result in emergent patterns that stabilize in unpredictable but orderly ways
Results
